VAANIX: A Real-Time Voice AI Agent Platform with Action-Enabled Agents, Automated Appointment Booking and a Business Workspace

Abstract

Telephone support is still the main way many small businesses talk to their customers, yet it is also the channel they are least able to staff around the clock. Traditional IVR menus frustrate callers, and most of the newer AI voice products are closed, priced per minute and difficult to adapt to a specific business. This paper presents VAANIX, a self-hostable voice agent platform that connects ordinary phone calls to a streaming pipeline of speech recognition, a large language model and speech synthesis, so that a caller can simply talk and be understood. The first version of the platform focused on that conversational pipeline and a developer console. In this paper we describe a second, larger iteration. A single Agent Studio now holds everything about an agent: its prompt, voice, knowledge base, API settings, call logs, single calls and bulk calling. Agents can also be given actions at creation time. The most important of these is appointment booking, in which the agent checks availability while the caller is still on the line, books the slot, and confirms it by voice. Confirmed bookings appear immediately on a Business Workspace home screen that shows a calendar of the client\'s own appointments instead of generic call statistics. We explain the architecture, the action mechanism, the way double bookings are prevented, and the security design built on row level isolation. Manual testing across all modules showed that callers receive a spoken reply typically within one to two seconds and that booked slots are reflected on the calendar within moments, without the per-minute platform fees charged by managed voice agent services.

Introduction

The text introduces VAANIX, an AI-powered voice agent system designed to automate phone calls and appointment booking for clinics and small businesses.

Traditional phone systems often rely on touch-tone menus that frustrate callers and cannot understand natural speech. Although modern AI models can communicate naturally, creating a real-time voice assistant requires low-latency speech recognition, language processing, and speech synthesis.

The first version of VAANIX connected Twilio, Deepgram, Claude, and ElevenLabs to create a voice agent that could answer callers but could not perform real tasks. The second version addresses this limitation by introducing three key features:

• Agent Studio: A centralized interface for creating and configuring AI voice agents, connecting phone numbers, and reviewing call logs.

• Action Mechanism: Allows the AI agent to execute backend functions, such as checking appointment availability and booking appointments.

• Business Workspace: Provides a practical dashboard, including a calendar for viewing booked and scheduled appointments.

The system uses a four-layer architecture: telephony, real-time voice processing, action and business logic, and data storage and presentation. It integrates Twilio for telephone communication, Deepgram for speech recognition, Claude for language understanding and tool calling, ElevenLabs for speech synthesis, Supabase PostgreSQL for data storage, and React 19 for the user interface.

The architecture separates time-sensitive voice processing from slower operations, such as booking appointments and updating calendars, to maintain responsive conversations.

The paper also reviews related research on voice agents, language-model tool use, speech technologies, and knowledge retrieval. Its main contribution is bridging the gap between conversational AI and practical business operations by enabling configurable agents to perform real actions and display their results in a business-focused interface.

Conclusion

VAANIX started as a talking pipeline and has become a platform that gets things done on a phone call. The Agent Studio puts every part of an agent in one place, the action mechanism gives each agent only the capabilities it needs, and the Business Workspace shows the client the outcome in the form they care about, a calendar of real appointments. Two design decisions turned out to matter most. The first was to enforce the no-double-booking rule in the database and not in the conversation. The second was to attach identity and ownership on the server and never leave them to the language model. Both are cheap to implement and make the system far easier to trust. The work also showed that most of the effort in a voice agent goes into the edges of the conversation, such as interruptions, spoken dates and silent moments, and not into the model itself. We believe a small, self-hosted platform of this kind is a realistic option for clinics, salons, coaching classes and other appointment driven businesses that cannot afford a full call centre but do not want to lose calls.

References

[1] J. Qiu, Z. Chen, L. Yang, M. Zhu, Z. Liu, J. Tan, W. Zhao, R. Murthy, R. Ram, A. Prabhakar, S. Heinecke, C. Xiong, S. Savarese, and H. Wang, \"Building Enterprise Realtime Voice Agents from Scratch: A Technical Tutorial,\" arXiv:2603.05413, Mar. 2026. [2] Twilio, \"Media Streams,\" Twilio Voice Documentation. [Online]. Available: https://www.twilio.com/docs/voice/media-streams [3] Deepgram, \"Streaming Speech-to-Text,\" Deepgram Developer Documentation. [Online]. Available: https://developers.deepgram.com/docs [4] ElevenLabs, \"Text to Speech,\" ElevenLabs Documentation. [Online]. Available: https://elevenlabs.io/docs [5] Anthropic, \"Tool use with Claude,\" Claude API Documentation. [Online]. Available: https://docs.claude.com [6] Supabase, \"Row Level Security,\" Supabase Documentation. [Online]. Available: https://supabase.com/docs/guides/database/postgres/row-level-security [7] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, \"ReAct: Synergizing Reasoning and Acting in Language Models,\" in Proc. International Conference on Learning Representations (ICLR), 2023. [8] T. Schick, J. Dwivedi-Yu, R. Dessi, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom, \"Toolformer: Language Models Can Teach Themselves to Use Tools,\" in Advances in Neural Information Processing Systems (NeurIPS), 2023. [9] P. Lewis et al., \"Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,\" in Advances in Neural Information Processing Systems (NeurIPS), 2020. [10] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, \"Robust Speech Recognition via Large-Scale Weak Supervision,\" in Proc. International Conference on Machine Learning (ICML), 2023. [11] T. Stivers et al., \"Universals and cultural variation in turn-taking in conversation,\" Proceedings of the National Academy of Sciences, vol. 106, no. 26, pp. 10587-10592, 2009. [12] ITU-T, \"Recommendation G.114: One-way transmission time,\" International Telecommunication Union, 2003. [13] B. Desruisseaux, Ed., \"Internet Calendaring and Scheduling Core Object Specification (iCalendar),\" RFC 5545, Internet Engineering Task Force, Sep. 2009.

Copyright

Copyright © 2026 Prof. Saba Chaughule, Atharv Bhosale, Srushti Phalke, Srushti Magar, Avdhoot Chaudhari. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.